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Executive summary 


The project 


The Tutor Trust is a Manchester-based charity that aims to provide affordable small group and one-to- 
one tuition, predominantly to disadvantaged pupils in schools in challenging communities. The tutors 
are university students and recent graduates, enabling tuition to be provided at a competitive rate on 
a not-for-profit basis. 


This three-year evaluation assessed the impact of the Tutor Trust on the English and mathematics 
GCSE results of 1,029 Year 11 pupils. Tuition took place in the academic years 2011-2012 to 2013- 
2014. Students received different numbers of hours of tuition, and some received tuition in multiple 
academic years between Year 9 and Year 11. The evaluation also explored schools’ perceptions of 
the need for affordable tuition and their assessment of the quality of the service provided. 


Qualitative fieldwork took place in eight schools and was based on interviews with senior leaders, 
classroom teachers, tutors and pupils. 


The project was funded by the Education Endowment Foundation (EEF), Manchester City Council 
and payments from participating schools. A separate evaluation of the impact of the tuition on the 
English and mathematics attainment of Year 6 and Year 7 pupils is available on the EEF website. 


Key conclusions 

1. Due to the limitations of the study design and the absence of a high-quality comparison 
group, this evaluation has not provided a secure estimate of the impact of the project on 
academic outcomes. 

2. Participating pupils achieved slightly higher mathematics GCSE scores than pupils in the 
comparison group, and lower English GCSE scores than pupils in the comparison group. 
However, it is not possible to attribute either change to the tuition provided. 

3. Schools involved in the qualitative interviews were positive about the tuition, keen to work 
with the Tutor Trust again, and largely confident that the tuition was beneficial for their 
pupils. 


4. Teachers believed that there was a need for more affordable high-quality tuition, and that the 
Tutor Trust helped meet this need. 


5. It is recommended that the Tutor Trust continues to monitor tutor performance and identifies 
mechanisms to increase the consistency of tuition. Schools should be active participants in 
this process and classroom teachers need to be involved in the planning and management 
of tuition to ensure that in all cases tuition complements work in the classroom. 


Ss it ti Security rating awarded as part of 
ecurity rating the EEF peer review process 


Overall, the findings from this evaluation have very low security. The evaluation was set up as an 
effectiveness trial, meaning that it aimed to test the approach under realistic conditions in a large 
number of schools. Impact data were obtained from 24 schools for mathematics and 19 schools for 
English. However, limitations of the evaluation design and observable differences between the 
students in the intervention and comparison groups meant that it was not possible to establish a high- 
quality comparison group against which to assess the impact of the tuition provided. Consequently, 
the findings are deemed too insecure to assess the impact of the programme on pupil attainment. 


The evaluation used a quasi-experimental design, which aimed to enable a comparison to be made 
between participating pupils and other pupils who were similar in terms of their demographic and 
socio-economic characteristics. However, it was not possible to control for a number of potentially 
influential variables, which the evaluators had planned to do. These variables were pupil-level special 
educational needs (SEN); English as an additional language (EAL); ethnicity; and looked after 
children (LAC) status. They were excluded from the final analysis due to a change in the level of 
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consent required to obtain this data, which occurred after the data collection phase of the trial had 
been completed. 


Security was further weakened by differences in the prior attainment of pupils in these schools and 
those in comparison schools. Although some differences were accounted for through statistical 
analysis, the presence of such marked differences indicated that the comparison group was unlikely 
to be of a high quality. In addition, it was not possible to identify with confidence which pupils in the 
comparison group would have received tuition. 


Finally, it was not possible to take into account unobservable factors between pupils that received 
tutoring and those who did not, such as their motivation or the quality of the leadership in their 
schools. To overcome this weakness, an alternative design, such as a randomised controlled trial, 
would have been necessary. 


Results 


e In general, teachers in participating schools were positive about the tuition provided and a 
large majority of the senior leaders interviewed were keen to work with the Tutor Trust again. 

e Tuition was perceived to be most effective when tutors possessed strong pedagogical skills 
and subject knowledge; engaged and interacted successfully with pupils; and targeted 
sessions appropriately. 

e On average, pupils receiving mathematics tuition achieved slightly higher mathematics GCSE 
scores than pupils in the comparison group. On average, pupils receiving Tutor Trust English 
tutoring achieved lower English GCSE scores than pupils in the comparison group. However, 
it is not possible to attribute these differences to the tuition provided. Specifically, the absence 
of a high-quality comparison group made it impossible to rule out the possibility that the 
differences observed were due to differences between pupils who received tuition and those 
who did not. The quality of the comparison group was further compromised by being unable 
to use pupil-level information about SEN, EAL or LAC status in the evaluation. 


How much does it cost? 


The Tutor Trust charged £18 per hour of tuition in 2011-2012, irrespective of the size of the tutoring 
group. In 2012-2013, 1:1 tuition was charged at £18 per hour; 1:2 at £20 per hour, and 1:3 at £26 per 
hour. Based on groups of three pupils receiving 25 tuition sessions, the total cost is estimated at 
approximately £217 per pupil. 


Group Effect size Estimated Security rating Cost 
months’ progress 


All pupils +0.05 +1 months EEE 
(mathematics) 


All pupils -0.16 -2 months 
(English) 
FSM pupils +0.09 +1 months 


(mathematics) 


FSM pupils -0.03 -1 months 
(English) 
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Introduction 


Intervention 


The Tutor Trust is a charity based in Manchester that aims to provide affordable small group and one- 
to-one tuition to schools. It was established in 2011 following a £180,000 grant from the Education 
Endowment Foundation. The Trust recruits university students and recent graduates, which enables it 
to provide tuition at a competitive rate. It predominantly aims to support schools in challenging 
communities and pupils who are looked-after or eligible for free school meals. 


The Tutor Trust rigorously screens tutor applicants to ensure that they have the appropriate mix of 
academic skills, teaching ability, and interpersonal skills. It accepts around 30% of applicants. It also 
delivers structured training to equip tutors with the ability to: decide on a long-term tuition plan for 
pupils; assess pupils’ grades/levels; plan a tuition lesson tailored to the needs of the pupil (this 
includes using the seven-point tuition plan structure); manage pupil behaviour; give praise and 
encouragement effectively; identify a pupil’s preferred learning style(s) and alter tuition accordingly; 
and teach English and mathematics appropriately. The Tutor Trust also ensures that tutors have valid 
Disclosure and Barring Service checks and organises the tuition sessions. 


In this project, Tutor Trust tutors were paid to deliver tuition to pupils in Years 9, 10, and 11. All 
tutoring took place on the premises of the participating schools. Most schools chose to use small 
group tuition, rather than one-to-one tuition. The tuition was designed to meet the needs of individual 
schools and pupils—hence the format and content of sessions varied (though tutors were provided 
with a model on which to broadly base tuition sessions). The schools chose which pupils would 
receive tuition; there were no prescribed selection criteria. 


Tuition in Phase 1 took place in the Academic Year 2011-2012; in Phase 2 in the Academic Year 
2012-2013; and in Phase 3 during the Academic Year 2013-2014. According to data provided by 
participating schools, in Phase 1, 74 pupils received tutoring in English, and 242 received tuition in 
mathematics, one pupil received tutoring in both. In Phase 2, 275 pupils received tutoring in English, 
and 323 in mathematics, with 20 of these pupils receiving tutoring in both subjects. * In Phase 3, 199 
pupils received tutoring in English, 263 in mathematics and 140 received tutoring in both subjects. 


Background evidence 


The rationale for establishing the tuition programme was based on the effectiveness of private tuition 
according to previous academic studies. The evidence synthesised within the Sutton Trust—EEF 
Toolkit? suggests that one-to-one tuition, where a student receives intensive tuition, generates five 
months of additional progress on average, and that small group tuition generates four months of 
additional progress on average. The evidence base related specifically to tuition delivered by 
university students is weaker. However, a small number of US studies have found that programmes 
led by college students can lead to improvements in student outcomes (Elbaum ef al., 2000), 
suggesting that this is an area worthy of further evaluation in the UK. 


The EEF commissioned this independent evaluation to establish whether the Tutor Trust is achieving 
its aims, and the extent to which the model is perceived to be sustainable, in relation to demand and 
delivery of low-cost tutoring in schools in Manchester. The evaluation was set up as an efficacy trial, 
meaning that it aimed to test the approach under ideal conditions in a moderate number of schools. 
The Trust has expanded since its inception, and in the four years it has been operating has delivered 


* Note that not all of the pupils who received tuition were included in the analysis. This is because when the data provided by 
the schools was matched to NPD data some pupils’ data could not be matched. This means that the numbers of pupils who 
received tuition does not match the numbers of pupils included in the analysis. 

? Available: http://educationendowmentfoundation.org.uk/toolkitione-to-one-tuition/ 
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over 40,000 hours of tuition to more than 8,000 pupils across over 100 primary and secondary 
schools. It has 500 tutors, who work in all ten Greater Manchester local authorities. 


Our evaluation focuses specifically on tuition of GCSE pupils. An efficacy study looking at the effect of 
the Tutor Trust’s tutoring on Year 6 and Year 7 pupils’ English and mathematics results is available 
separately on the EEF website. 


Evaluation objectives 
The central objective of the study was to address the question: 


e Do young people receiving tuition in English and mathematics obtain higher GCSE results 
than might otherwise be expected? 


The process evaluation focused on the model and its implementation, including the following: 


e Is the service fulfilling an identified need (i.e. lack of high-quality affordable tuition)? 
e Why do some schools decide not to take up the offer of tuition? 

e Are the tutors ‘high quality’? 

e Are the tutors delivering the tuition sessions in line with schools’ expectations? 

e Is the Tutor Trust meeting its recruitment targets? 


Project team 


NFER directed and led the process and impact evaluations. Helen Aston was Project Director. Emily 
Buchanan led the process evaluation with the support of Matthew Walker. Jo Morrison led the impact 
evaluation with the support of Rose Cook. Nick Bent, Director of the Tutor Trust was responsible for 
gaining schools’ consent to participate in the evaluation and collecting specified pupil data. 


Ethical review 


The evaluation was delivered in accordance with NFER’s Code of Practice. The Tutor Trust secured 
schools’ consent to participate in the evaluation (see Appendix 1). Additionally, NFER sought and 
secured consent from all interview participants to collect and use their anonymised data (see 
Appendix 2). Given the age of the pupils, it was deemed appropriate for head teachers to act in loco 
parentis and to give their consent for pupils’ involvement. 
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Methodology 


Impact evaluation design 


The evaluation used a quasi-experimental design to evaluate the impact of tutoring on test scores. 
Given the strong existing evidence base related to tuition (particularly one-to-one tuition) this design 
was judged to be appropriate. In essence, the evaluation was designed to enable the evaluation team 
to check whether the effects were broadly similar to those found in previous studies. However, as 
there was no randomisation at either the school or pupil level, there are limitations (discussed further 
below) to the extent that we can interpret the measured differences in test scores as the causal 
impact of tutoring. 


The Tutor Trust recruited schools to the evaluation. The Trust was particularly keen to engage 
schools in deprived communities in Greater Manchester,* and contacted schools directly to publicise 
the tuition and to encourage schools to buy into the service. The intervention group was therefore 
self-selecting. 


The NFER team designed pupil data forms which were administered to schools by the Tutor Trust (an 
example is provided in Appendix 3). These forms asked schools to provide information on the tuition 
in their school, including pupil names and background characteristics, tuition subject, number of 
tuition sessions, and whether or not tutees missed lessons for their tuition. The analysis was based on 
the information provided in these forms, matched with additional pupil-level data from the National 
Pupil Database. 


The comparison group was formed of pupils in the same year group from schools within Manchester’s 
statistical neighbours.* To enable comparison with similar pupils, we controlled for available 
background factors in a multilevel regression model.” We controlled for prior attainment (KS3 Teacher 
Assessment level in English or mathematics); gender; and free school meal (FSM) eligibility. The 
analysis also controlled for school-level factors: school type; percentage of pupils in the school who 
are eligible for FSM; percentage of pupils gaining 5+ GCSEs at grade A*—C; size of the Key Stage 4 
cohort; percentage of pupils in the school with SEN; number of full-time equivalent teachers; and 
schools’ average of the pupils’ total GCSE point score. Note that in a deviation from the original 
planned analysis, we were unable to control for a number of potentially influential variables. These 
comprised: pupil-level special educational needs (SEN); English as an additional language (EAL); 
ethnicity; and looked after children (LAC) status. These variables were excluded from the final 
analysis due to a change in the level of consent required by the NPD in order to release this data, 
after we had completed our data collection in schools. 


Unlike in a randomised controlled trial, we cannot be confident that unobserved background 
characteristics that are a factor in schools’ decisions to nominate pupils for tutoring (Such as a pupil 
being particularly motivated to learn), or the unobserved characteristics of the schools the pupils 
attend, are balanced between the intervention and the comparison group at baseline. This means that 
our estimates of the impact have to be interpreted carefully: any differences in outcome could also be 
the result of underlying differences that were not taken into account in selecting a comparison group 
because we had not measured those characteristics. 


3 The Tutor Trust did not define ‘deprived communities’, meaning that there were no strict inclusion criteria. 


“ ‘Statistical neighbours’ are local authorities which have similar characteristics. Manchester's statistical neighbours are: 
Nottingham, Greenwich, Liverpool, Barking and Dagenham, Birmingham, Salford, Middlesbrough, Kingston upon Hull, 
Southampton, and Newcastle upon Tyne. 

° To explore if tuition was associated with improved GCSE results, the study assumed that GCSE results were related to a set 
of variables, including tuition, and estimated the relationship between tuition and GCSE scores. 


Education Endowment Foundation 7 


Tutor Trust: Secondary 


Eligibility 


The Tutor Trust sought to recruit secondary schools in Manchester to receive the tutoring. Any 
secondary school in Manchester was eligible to receive tuition and the sample was self-selecting at 
the school level. A core part of the model was that schools had freedom to select whichever pupils 
they wanted to receive tuition. There were therefore no pupil-level eligibility criteria. We asked schools 
to identify why they had selected individual pupils to receive tutoring from a list of options (see 
‘Participants’ section in Results). 


The Tutor Trust secured schools’ consent to participate in the evaluation (see Appendix 1). 
Additionally, NFER sought and secured consent from all interview participants to collect and use their 
anonymised data (See Appendix 2). Given the age of the pupils, it was deemed appropriate for head 
teachers to act in loco parentis and to give their consent for pupils’ involvement in interviews. 


Intervention 


University pupils recruited and trained by the Tutor Trust delivered tuition to pupils in Years 9, 10, and 
11. Our evaluation was concerned primarily with tuition delivered to GCSE students; Year 9 tuition is 
outside the scope of this study. 


All of the tutors involved in the programme had to pass a two-stage recruitment process administered 
by the Tutor Trust. The process was designed to ensure that tutors had the appropriate mix of 
academic skills, teaching ability and interpersonal skills. The first round of recruitment consisted of a 
written, online application form modelled on the Teach First application form and designed in 
consultation with Teach First and Pricewaterhouse Coopers (PwC). There were also minimum 
qualification requirements: tutors had to have at least an ‘A’ at GCSE (or equivalent or higher) in the 
subject(s) they wished to tutor in. Additionally, tutors were asked to detail their previous experience 
working with children, to provide a personal statement explaining their motivation for working with the 
Tutor Trust, and to answer competency-based questions looking at humility, resilience, planning and 
organisation, and creativity. The second round of recruitment involved a short face-to-face interview 
with Tutor Trust staff to probe applicants’ motives, experience, dependability and communication and 
interpersonal skills. 


Tutors received a minimum of two-and-a-half days of structured and unpaid training before starting 
tutoring. The training sessions comprised: 


e A three-hour evening session on safeguarding, professionalism, why tuition matters and 
working with the Tutor Trust. The session was led by Tutor Trust staff. 

e A full-day session on lesson planning, learning styles, behaviour management, varying 
lessons, differentiation, creating a scheme of work for a tuition assignment and progress 
tracking across an assignment with the pupil. An educational consultant with extensive 
teaching experience led this session. 

e A full-day session on teaching the relevant subject to pupils, delivered by educational 
consultants who were commonly current or former teachers or senior leaders. The focus was 
on subject-specific tasks; curriculum understanding; common mistakes and misconceptions 
pupils may hold; teaching tools; assessing the level that pupils are working at; and selecting 
suitable tuition material. 


Tutors were also given access to additional CPD sessions; a range of Tutor Trust online resources; 
regular opportunities to interact informally and share problems and solutions with each other; and 
support from Tutor Trust staff as needed. All tutors were introduced to a seven-point tuition plan to 
structure tuition sessions (introduction, remember, model, try, apply, secure, reflect) during their 
training, but encouraged to deviate from it as appropriate. Hence, the nature and content of tuition 
sessions varied with the needs of individual schools and pupils. 


Education Endowment Foundation 8 


Tutor Trust: Secondary 


Table 1 below sets out the periods during which tuition was provided. All tutoring took place on the 
premises of the participating schools. Schools chose to purchase small group or one-to-one tuition. 
The tuition was designed to meet the needs of individual schools and pupils, rather than following a 
set format. Pupils typically received tuition on a weekly basis and tuition sessions lasted 
approximately one hour. 


Table 1: Tuition delivery periods 


Activity Delivery periods 
Phase 1 tuition February to June 2012 (Academic Year 2011-12 


Phase 2 tuition September 2012 to July 2013 (Academic Year 2012-13) 
| Phase 3 tuition September 2013 to July 2014 (Academic Year 2013-2014) 


Outcomes 


We selected English and mathematics GCSE results as the outcome measures for the study because 
the tuition is intended to aid GCSE performance in these subjects. GCSE data for participating pupils 
and other pupils, from which a comparison group could be selected, is also available on the National 
Pupil Database (NPD). 


Sample size 


We calculated the power of the proposed statistical analysis at the outset using estimated pupil 
numbers. This is because it was not possible to accurately predict the actual numbers of pupils who 
would be receiving Tutor Trust tuition in subsequent academic years. As the evaluation has been 
rolled out incrementally for three years/phases, our calculations are for the overall evaluation i.e. 
using estimated sample size of year 11 pupils for the full three years. Using an average of 25, 27, and 
30 GCSE-taking pupils per school, in 50 schools, ° and assuming a comparison group of an equivalent 
size, an estimated intra-class correlation of 0.1 and a correlation between prior attainment (KS3) and 
outcome (GCSE) measures of 0.6, the power of the analysis to detect an effect size of 0.2 at the end 
of the study was greater than 90% (see Figure 1 below). 


Taking into account the overlap in schools between the three phases, we have data from 31 schools. 
However, following matching to the NPD, data from 781 pupils and 24 schools remained in the 
dataset for mathematics. The English analysis included data from 409 pupils in 19 schools. The 
achieved sample size and power are therefore lower than planned. 


Since tutoring is a pupil-level intervention, and not a whole-school intervention, it is not straight- 
forward to calculate the minimum detectable effect size (MDES) at 80%. The model contained 
comparison pupils both from Tutor Trust schools and non-Tutor Trust schools, meaning that the 
intervention varied at both the pupil and school level. We calculated the MDES at the point of 
analysing the data as if the intervention were a whole school intervention, giving a conservative 
estimate. The minimum detectable effect size was 0.45. 


° The initial intention was for the Tutor Trust to deliver tutoring in 50 schools in the third year of this study. 
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Figure 1: Power calculations 
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Analysis 


We carried out multilevel modelling, a development of a common statistical technique known as 
‘regression analysis’. This looks at the relationship between the ‘dependent variable-—GCSE 
outcome—and a number of ‘independent variables’ including, in particular, a variable which indicated 
whether or not a pupil was in a school involved with Tutor Trust and was nominated for tutoring. 
Multilevel modelling is one of a number of techniques that can be used to take account of the 
clustering of the data—i.e. individual pupils being grouped within local authorities and schools—to 
produce accurate estimates of the differences between pupils, schools, and local authorities. 


Multilevel modelling allowed us to identify any differences in performance between Tutor Trust pupils 
and the comparison pupils (similar pupils in similar schools) in their GCSE mathematics and English. 
The analysis identified the difference in GCSE points, whilst controlling for the influence of pupil 
characteristics known to affect GCSE performance that were available on the NPD and available 
school-level characteristics. 


We controlled for prior attainment (KS3 Teacher Assessment level in English or mathematics); 
gender; and free school meal (FSM) eligibility. The analysis also controlled for school-level factors: 
school type; percentage of pupils in the school who are eligible for FSM; percentage of pupils gaining 
5+ GCSEs at grade A*-C; size of the Key Stage 4 cohort; percentage of pupils in the school with 
SEN; number of full-time equivalent teachers; and the school’s average of the pupils’ total GCSE 
point score. Note that we were unable to control a number of potentially influential variables due to the 
sensitivity of the data. These were pupil-level special educational needs (SEN), English as an 
additional language (EAL), ethnicity, and looked after children (LAC) status. 
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It is important to note that the analysis only accounted for the characteristics of pupils that were 
captured in the data, and so it is possible that any differences identified between participating pupils 
and other similar pupils were caused by other unobserved factors. 


Our analysis was done on an ‘intention to treat’ basis.’ 


In line with EEF guidance, the effectiveness of tutoring for pupils who are eligible for FSM was 
assessed using subgroup analysis. 


Process evaluation methodology 


The process evaluation focused on Year 11 tuition. We collected qualitative data from eight 
secondary schools that received tuition over the three years/phases of the evaluation. Three schools 
were revisited over the three years/phases of the evaluation to explore changes over time. All of the 
case-study schools were serving challenging communities in the Greater Manchester area. We 
selected schools that had received the most hours of tuition (based on data provided by the Tutor 
Trust) as we felt that they would be better placed to provide rich data on the implementation of the 
tutoring and its perceived impact than those schools which had only received a small volume of 
tuition. We also sought to include schools with a range of different characteristics, including pupil 
gender, type of school, and local catchment. In Phase 3, we avoided schools who were participating 
in the separate evaluation of the Tutor Trust transition programme. NFER researchers collected all of 
the data. The Tutor Trust was not involved in the selection process and was not told which schools 
participated. Interviewees were selected in discussion with the schools. Schools were asked to 
identify pupils for interview who exhibited a broad range of characteristics and experiences (as 
relevant to each school). 


Interviews were carried out with senior school leaders; English and mathematics teachers, pupils who 
had received tutoring, and tutors. The interviews were carried out in the summer term of each of the 
three phases of the evaluation (2012, 2013, and 2014).°® The interviews (largely carried out face-to- 
face) allowed us to explore all aspects of the tuition in depth, and to gain a school- and pupil-level 
perception of the tutoring and its effectiveness. 


All of the case-study schools were offering tuition to Year 11 pupils in both English and mathematics. 
A full breakdown of the number of schools and interviewees involved in each phase of the evaluation 
is provided in the Table 2. 


’ This means that the analysis is based on the initial intentions to provide tuition, rather than whether or not pupils actually 
received tuition. So, if a pupil was intended to receive tuition, but for some reason did not, they would still be included in our 
analysis. 

® We are able to comment on three years of impact data in this Phase 2 report as the tuition had taken place by the time of 
reporting. We will not get access to the impact data for Phase 3 until spring 2015. This will be included in the Phase 3 report. 
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Table 2: Breakdown of interviewees in the process evaluation 


Phase 1 Phase 2 Phase 3 Total 
(2012) (2013) (40 )) 
Number of schools 4 4 3 11 separate 
(2 visited in (1 visited in Phase 2) interactions 
Phase 1) (8 different 
schools 
Number of interviews with 4 4 3 11 
senior school leaders 
Number of interviews with 4 3 2 9 
English/mathematics 
teachers 
Number of interviews with 20 5 3 28 
pupils (1 with a Year 10 pupil) 
Number of interviews with 4 4 5 13 
tutors 


The interviews with senior managers and classroom teachers covered the following topics: 


e contextual data on the school and its intake 

e how the tuition was implemented in school (e.g. number, frequency, duration, and length of 
sessions, reasons for offering tuition and how pupils were identified, tutor—pupil ratios) 

e whether tuition took place as originally planned 

e satisfaction with the quality of tutors and tuition 

e satisfaction with the service provided by the Tutor Trust 

e suggested improvements 

e impacts on pupils 

e whether the tuition would be beneficial in other schools and other areas. 


Interviews with pupils covered: 


e their experience of tutoring (e.g. whether they enjoyed it, what they thought of the tutors and 
the sessions) 

e whether the tuition was beneficial or not (and the reasons behind their answers) 

e if the tuition could be improved. 


Interviews with tutors addressed: 


e motivations for becoming a tutor 

e the training and support received from the Tutor Trust 

e whether they felt any improvements could be made to the training and support 
e their experiences of tutoring 

e impact on their tutees. 


Subject to the permission of participants, interviews were recorded and interviewers took handwritten 
notes. Interviews were summarised for each question (including verbatim quotes). The data was 
analysed using thematic codes in relation to each question/section of the interview schedule. The 
qualitative fieldwork was relatively small in scale and the sample size was insufficient to analyse data 
by type of tuition delivered or received, which was not an aim of the evaluation. 


We also collected data from schools that had declined the offer of working with the Tutor Trust. We 
collected responses via email and telephone from four schools in 2012, two schools in 2013, and four 
schools in 2014. The responses explored, what, if anything, would have made the Tutor Trust’s offer 
more attractive to these schools. 
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Costs 


The research team did not collect cost information; rather the tuition was delivered to schools using a 
fixed-price model. The Tutor Trust reported that it charged £18 per hour of tuition in Phase 1, 
irrespective of the size of the tutoring group. In Phases 2 and 3, the tuition rates changed according to 
the number of pupils in a tutoring session: one-to-one tuition was charged at £18 per hour; one-to-two 
at £20 per hour, and one-to-three at £26 per hour. Based on groups of three pupils receiving 25 tuition 
sessions, the total cost of the intervention is estimated as approximately £217 per pupil. 
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Impact evaluation 


Timeline 


In Phase 1, pupils received tutoring between February and June during the Academic Year 2011- 
2012. In Phase 2, tutoring took place between September 2012 and July 2013. In Phase 3, tutoring 
was delivered between September 2013 and July 2014. The Tutor Trust recruited schools in the run 
up to, and during, each of the Academic Years during which they delivered tuition. The evaluation of 
the programme ran from February 2012 to April 2015. 


Participants 


The Tutor Trust recruited schools through: directly approaching schools via letter, email, and 
telephone; word-of-mouth recommendations; repeat customers; and through Tutor Trust promotion at 
local-authority-wide events. The reasons given by schools for nominating the Year 11 pupils to 
receive mathematics and English tuition are set out in Tables 8 and 9 below.® 


In Phase 1, pupils received 1,404 mathematics and 482 English tuition sessions delivered by Tutor 
Trust tutors to groups of two to three pupils across the participating schools. The number of 
mathematics and English tuition sessions delivered per pupil ranged from one to 13 and one to ten 
respectively. On average, each mathematics tutee received 6.1 hours of tuition and each English 
tutee received 6.2 hours. 


In Phase 2, 4,159 mathematics and 1,817 English tuition sessions were delivered by Tutor Trust 
tutors to groups of two to three pupils across the participating schools. The number of mathematics 
and English tuition sessions delivered per pupil ranged from one to 30 and one to 15 respectively. On 
average, each mathematics tutee received 12.8 hours of tuition and each English tutee received 7.6 
hours. 


In Phase 3, pupils received 2,478 mathematics and 3,056 English tuition sessions. The number of 
mathematics and English tuition sessions delivered per pupil ranged from 1 to 28 and 1 to 58 
respectively. On average, each mathematics tutee received 11.7 hours tuition and each English tutee 
received 12.7 hours. See Appendix 4 for further information. 


On average across the three phases, more pupils received mathematics than English tuition. 
Participant flow 


Figures 2 and 3 give the numbers of Tutor Trust and comparison pupils included in the analysis of 
pupils receiving mathematics and English tutoring, respectively. 


There was no randomisation. 


. English pupils and non-Year 11 mathematics tutees were not included in the analysis in Phase 1, so we have not included 
data relating to them in this table. 
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Figure 2: Participant flow for pupils receiving tutoring in mathematics 


Tutor Trust pupils receiving tutoring in mathematics 
Participating in tutoring (n pupils = 1,004,*° n schools = 27) 
+ Excluded (n = 223) 


"not matched to NPD (n = 164) 
= missing GCSE mathematics grade (n = 0) 
"missing KS3 data (n = 59) 


Analysed (n pupils = 781, n schools = 24) 
FSM subset (n pupils = 220, n schools = 21) 


Comparison group pupils in mathematics 


NPD data for Manchester's statistical neighbours (n = 124,964) 


———_» Excluded (n = 23,973) 


"missing GCSE mathematics grade (n = 8,976) 
" missing KS3 mathematics level (n = 11,337) 
"missing school background data (n = 3,660) 
Analysed (n = 100,991) 
FSM subset (n = 27,125) 


*° We removed any duplicates from the dataset when the data provided by schools was matched to the National Pupil 
Database. Therefore, within the 781 pupils included in the analysis, no pupils are counted twice. However, a small number of 
the 223 pupils excluded from our analysis may have been counted twice. This is because some schools listed pupils with their 
UPN and again with their name. 
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Figure 3: Participant flow for pupils receiving tutoring in English 


Tutor Trust pupils receiving tutoring in English 
Participating in tutoring (n pupils = 765, n schools = 24) 
———» Excluded (n = 2777") 


"not matched to NPD (n = 208) 
"missing GCSE English grade (n = 15) 
"missing KS3 data (n = 54) 


Analysed (n pupils = 409, n schools = 19) 
FSM subset (n = 123, n school = 16) 


Comparison group pupils in English 
NPD data for Manchester's statistical neighbours (n = 80,034) 
Excluded (n = 17,064) 


"missing GCSE English grade (n = 8,438) 
"missing KS3 English level (n = 5,837) 
"missing school background data (n = 2,793) 


Analysed (n = 62,970) 
FSM subset (n = 16,706) 


The characteristics of the participating secondary schools are provided in Table 3. They are 
compared with the average respective characteristics of all secondary schools in England. The Tutor 
Trust secondary schools have more pupils eligible for FSM than the average school in England (58% 
of Tutor Trust secondary schools fall into the highest 20% of schools for their FSM eligibility, 
compared with 20% of secondary schools in England). However, the Tutor Trust secondary schools 
had similar levels of GCSE attainment to all secondary schools in England—there were no significant 
differences in the proportions of schools at each level of attainment. 


There was a much larger percentage of academies among the Tutor Trust schools than in all 
secondary schools in England (50% of Tutor Trust secondary schools were academies, compared 
with 11% of secondary schools in England“’). The Tutor Trust schools were also more likely to be 
located in urban areas (96% compared to 73% of secondary schools in England). Tutor Trust 
secondary schools were more likely to be rated by Ofsted as ‘requiring improvement’ or ‘inadequate’. 


“t We removed any duplicates from the dataset when the data provided by schools was matched to the National Pupil 
Database. Therefore, within the 409 pupils included in the analysis, no pupils are counted twice. However, a small number of 
the 277 pupils excluded from our analysis may have been counted twice. This is because some schools listed pupils with their 
UPN and again with their name. 


rs These figures related to 2013. 
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Table 3: Characteristics of participating schools 


All secondary Tutor Trust 


Tod Tete) om (2) secondary 

Tod aYote) [= (71) 

GCSE total point score Lowest 20% 36 38 
(quintile) 

2nd lowest 20% 16 15 

Middle 20% 16 19 

2nd highest 20% 16 8 

Highest 20% 16 19 

Free school meals Lowest 20% 20 4 
(quintile) 

2nd lowest 20% 20 0 

Middle 20% 20 12 

2nd highest 20% 20 27 

Highest 20% 20 58 

School type Community school 41 23 

Voluntary aided 16 15 

Voluntary controlled 10 0 

Academy 11 50 

Other 21 12 

Urban Rural 27 4 

Non-rural 73 96 

Ofsted rating Outstanding 20 4 

Good 59 48 

Requires improvement 19 32 

Inadequate 2 16 

Number of secondary schools 24,991 26 


Note: totals may not sum to 100% due to rounding. Chi-squared test of significance, p-values: GCSE quintile = 0.82, FSM 
quintile = 0.00, school type = 0.00, urban/rural = 0.02, Ofsted rating = 0.00. 


Intervention delivery 


Table 4 sets out the proportion of pupils who received individual, small group, or large group tuition 
and were included in our Phase 3 analysis. A tutor—pupil ratio of 1:3 was the most common choice in 
this cohort. ** 


*8 Note that the information in this table relates to the number of pupils and schools in the final matched dataset. The number of 
schools in the final modelling is different due to missing data on model variables. 


“ Note that we did not collect this data for Phases 1 and 2. We began to collect the data in Phase 3, because some of the 
earlier evaluation evidence had pointed to some tutors working with larger groups of pupils than intended. 
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Table 4: Tutor: pupil ratios in Phase 3 


BRU Co) ee oleloyime-tare) Mathematics English (%) 
(29) 


No information given 


1:1 


1:5 
1:6 0 0 
N pupils 403 339 


Table 5 summarises the numbers of pupils involved in tutoring across the three phases of the 
intervention. *° 


Table 5: Numbers of pupils receiving tuition 
Phase 1 Phase 2 Phase 3 All phases 


English only 76 328 199 603 
| Mathematics only 249 330 263 842 
| Both English and mathematics 1 21 140 162 


Tables 6 and 7 provide further information about English and mathematics tuition delivery, based on 
the data provided by the schools who participated in the evaluation. In Phase 2, the volume of tuition 
delivered increased, as was intended. The data we received indicated that there was considerably 
more mathematics tuition than English tuition. In the Phase 3 dataset, however, there were fewer 
mathematics sessions than in Phase 2, while the number of English sessions delivered increased 
again. Overall, 8,041 mathematics and 5,355 English tuition sessions were delivered. On average, 
pupils in receipt of English tuition received 8.8 hours of tuition, while those in receipt of mathematics 
tuition received slightly more than this: 10.2 hours on average. In Phase 1 all English tuition took 
place outside of lessons,*© but in Phases 2 and 3 around 40% of pupils missed lessons to attend 
tuition. For mathematics, the proportion of pupils missing lessons was slightly lower across all phases: 
around a third. 


*S These numbers differ to those used in the analysis due to the matching process with NPD data. Further, some tutees appear 
in more than one phase. 


*® Note that we did not collect data about which lessons pupils missed in order to attend tuition sessions. 
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Table 6: Mathematics tuition sessions 
Phase 1 Phase 2 Phase 3 All phases 


Total number of tuition sessions 1,404 4,159 2,478 8,041 
delivered 

Average number of sessions per 6 (1-13) 13 (1-30) 11 (1-28) 10 
pupil (and range)”’ 

Average number of tuition hours per 6.1 12.8 11.7 10.2 
pupil 

Number of pupils missing lessons to 90 119 125 334 


attend tuition 


Number of pupils attending tuition 160 208 265 633 
outside of lessons 


Table 7: English tuition sessions 


Phase 1 Phase 2 Phase 3 All phases 


Total number of tuition sessions 
delivered 


Average number of sessions per 6 (1-10) 8 (1-15) 11 (1-58) 
pupil (and range)” 


Average number of tuition hours 6.2 7.6 12.7 8.8 
per pupil 


Number of pupils missing lessons 0 113 94 207 
to attend tuition 


Number of pupils attending tuition 75 136 238 449 
outside of lessons 


Tables 8 and 9 set out the reasons why schools selected pupils to receive tuition. Pupils were mainly 
selected for tuition if they were at lower levels of predicted GCSE attainment. The majority of all pupils 
and of pupils eligible for FSM were selected for mathematics and English tuition because they were 
either at the grade D/C borderline or struggling to attain grade DIC. It is interesting that for the English 
tuition, compared to mathematics tuition, a greater proportion of pupils were selected for reasons 
other than the three options concerning their predicted attainment at GCSE.” 


’ The range shows the minimum and maximum number of sessions received by pupils within the relevant phase, based on the 
data schools provided to the Tutor Trust. Further information on the number of sessions received by pupils is available in 
Appendix 4. 


*® The range shows the minimum and maximum number of sessions received by pupils within the relevant phase, based on the 
data schools provided to the Tutor Trust. Further information on the number of sessions received by pupils is available in 
Appendix 4. 


a We did not collect data on what the ‘other’ reasons were. 
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Table 8: Reasons given by schools for selection of Year 11 mathematics tutees 


All pupils Pupils eligible for FSM 


Reasons for receipt of mathematics tuition N % N % 


| 

Potential to get grade B or above 56 6.6 11 4.6 
| At grade DIC borderline, aiming for grade C 449 53.2 114 48.1 
Struggling to attain grade DIC borderline 174 20.6 63 26.6 
Other (not specified) 101 12.0 28 11.8 
Missing 64 7.6 21 8.9 
Total” 844 100.0 237 100.0 


Table 9: Reasons given by schools for selection of Year 11 English tutees”* 


All pupils Pupils eligible for FSM 


Reasons for receipt of English tuition N % N % 
Potential to get grade B or above 35 5.2 6 4.2 
At grade DIC borderline, aiming for grade C 282 42.2 60 42.0 
Struggling to attain grade DIC borderline 146 21.9 48 33.6 
Other (not specified) 195 29.2 27 18.9 
Missing 10 1.5 2 1.4 
Total 668 100.0 143 100.0 


Pupils receiving tutoring were compared to other pupils from schools within Manchester's statistical 
neighbours. We do not know what other initiatives the comparison pupils have been involved in. It 
would be unreasonable to expect that they experienced no initiatives, so the analysis compares Tutor 
Trust tuition with what would otherwise have taken place. Some pupils missed lessons to receive 
tutoring, so for some pupils the tuition was in addition to their regular lessons while for others tuition 
was in place of regular lessons. Where pupils missed lessons to receive tuition, their classmates 
benefitted from smaller class sizes. However, the vast majority of the comparison group were in LAs 
that were not participating in Tutor Trust tutoring, where class sizes would remain unaltered. 


The process evaluation highlighted a number of fidelity issues. For example, tuition sessions were not 
always small group or one-to-one sessions. The interview data also revealed that some whole class 
sessions were delivered, but were not identified in the information about tutoring provided by the 


?° Note that the information in this table relates to the number of pupils in the final matched dataset. The number of schools in 
the final modelling is different due to missing data on model variables. 


21 Note that FSM data in this table is only for Phases 2 and 3, as English tuition data was not matched to NPD so this 
information is not available. 
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schools in the pupil data forms. Indeed, the findings from the process evaluation suggest the data 
provided by schools may not be an accurate record of the tuition that actually took place. This is 
discussed later in the report under the sub-heading ‘Limitations’. A further fidelity issue concerns the 
quality of tutors. The Tutor Trust programme is intended to provide high-quality tuition, but some 
schools from all three phases of the process evaluation have raised issues regarding tutor quality ina 
small number of cases. More information is provided under the sub-heading ‘Fidelity’ below. 


Pupil characteristics 
The characteristics of the pupils included in the impact analysis” are provided in Tables 10 and 11. 
_ Table 10: Demographic data for mathematics pupils 


Pupils nominated for Tutor Comparison group 
Trust mathematics tutoring 


Sex 
Boys 42 352 50 52,382 


Girls 58 488 50 52,285 
Eligible for FSM 
No 73 610 73 76,864 
Yes 27 230 27 27,803 
Missing 0 0 2 1,782 
School type 
Comprehensive 52 437 61 63,518 
Grammar 0 3 3 3,381 
Academy 48 400 34 35,440 
“Specialschool =  O 0 2  —-:1,698 
Other 0 0 0 0 
Single sex / Co-educational school 
Mixed 82 685 80 83,212 
Boys school 2 14 7 7,522 
Girls school 17 141 13 13,933 
Attainment Mean SD Mean SD 
Raw average GCSE points score 35.6 7.6 38.2 11.8 
Maximum GCSE points score 58 - 58 - 
(equivalent (equivalent to 
to an A*) an A*) 
Average KS3 points score 33.12 5.5 36.33 7.9 
Total 840 104,667 


72 Note that the modelling takes account of the differences as measured by these variables between Tutor Trust and 
comparison group pupils. 
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The differences between the average GCSE point scores and the average KS3 point scores for pupils 
receiving tutoring and the comparison group were tested using f-tests and were found to be 
statistically significantly different. 


Table 11: Demographic data for English pupils 
Pupils nominated for Tutor Trust 
English tutoring 


Comparison group 


% N % N 

Sex 

Boys 54 251 50 32,741 

Girls 46 212 50 33,208 

Eligible for FSM 

No 71 327 74 48,572 

Yes 29 136 26 17,377 

Missing 0 0 1 764 

School type 

Comprehensive 21 96 53 35,197 

Grammar 0 2 3 2,099 
| Academy 79 365 42 27,493 

Special school 0 0 1 850 

Other 0 0 0 282 

Single sex/Co- 

educational 

school 

Mixed 100 463 80 52,517 

Boys school 0 0 t 4,578 

Girls school 0 0 13 8,854 

Attainment Mean SD Mean SD 

Raw average 37.4 6.0 39.7 9.0 

GCSE points 

score 

Maximum GCSE 52.0 - 58.0 = 

points score 

Average KS3 33.9 4.4 35.0 6.2 

points score 

Total 463 65,949 


The differences between the average GCSE point scores and the average KS3 point scores for pupils 
receiving tutoring and the comparison group were tested using f-tests and were found to be 
statistically significantly different. 


While there are differences, as measured by these variables, between the pupils nominated for tuition 
and the comparison group, these are accounted for in the models used in the analysis (by including 
these characteristics as background factors we are comparing like with like). 
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Numbers analysed 


We included 781 Tutor Trust pupils from 24 schools in the analysis of mathematics GCSE results. For 
the analysis of English GCSE results, 409 Tutor Trust pupils from 19 schools were included. The 
respective comparison groups were made up of 100,991 pupils with mathematics GCSE results and 
62,970 pupils with English GCSE results from Manchester's statistical neighbours. 


The outcome measures were not imputed (Substituted with another relevant value, if missing). Pupils 
with missing mathematics or English GCSE outcomes were excluded from the respective models and 
are therefore not included in the numbers cited in the paragraphs above. In total across phases, 78 
pupils who were tutored in mathematics had a missing GCSE mathematics outcome and 15 pupils 
who were tutored in English had a missing GCSE English outcome. 


Where pupils had missing categorical background data (school or individual level), we coded these as 
missing and included these missing categories in the models. Where pupils had missing continuous 
background data (e.g. prior attainment, school-level FSM rates) their missing data was replaced with 
mean values. 


Outcomes and analysis 


We used multilevel regression models to analyse the mathematics and English GCSE grades of 
pupils who were nominated to receive tutoring (so including a small number of pupils who were 
nominated but did not go on to receive tuition) in mathematics and English. We compared their results 
to a ‘comparison’ group comprised of pupils in Manchester’s statistical neighbours. The multilevel 
models analysed the data at three levels—local authority, school, and pupil—taking account of the 
variation in GCSE point scores at each level. The models also took account of other background 
factors measured on the NPD including KS3 prior attainment, and also some school-level variables 
(school type, single sex/coeducational, school size, percentage of pupils eligible for free school 
meals, percentage pupils with EAL, percentage of pupils with any level of SEN, school average total 
GCSE point score and the percentage of pupils achieving 5 GCSE A*-C grades including English and 
maths). We also ran separate models based only on the subsets of pupils who were eligible for free 
school meals. 


Table 12 presents a summary of results from the impact analysis, summarising the number of pupils 
in the analysis and the estimated effect size with confidence intervals. 
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Table 12: Summary of impact analysis results 


Effect size Number of Number of 
(95% intervention comparison 


confidence pupils pupils 
interval) 

Mathematics tuition vs. comparison 0.051 100,991 
(0.001, 0.101) 

Mathematics tuition vs. comparison (FSM 0.090 27,125 

only) (-0.008, 0.188) 


-0.161 
(-0.291, -0.03) 


English tuition vs. comparison (FSM only) -0.026 123 16,706 
(-0.219, 0.168) 


English tuition vs. comparison 


Analysis of mathematics tutoring 
Tables 13 to 16 set out the results of the main impact analysis for mathematics tutoring. 


Table 13: Analysis of mathematics GCSE point score—base model fixed effects 


lexer-ynicey(-lalt Teslaletslaem=\ane) a 95% confidence 
interval 


Intercept (33.835, 35.745) 


Table 14: Analysis of mathematics GCSE point score—base model random effects 


WEVatclalexss pV eclalelslae msl age) @ 


LA 0.498 0.992 
School 56.411 4.602 
Pupil 117.072 0.511 


Table 15: Analysis of mathematics GCSE point score—final model fixed effects 


foxer-ynice(clal' PJrclaletslaems) age) ame bob more) avite (=) ales) 


interval 


Pupil-level variables 


Intervention 0.601 0.303 (0.02, 1.19) 
Average Key Stage 3 1.138 0.003 (1.13, 1.14) 
points—mathematics 

Eligible for free school meals -1.274 0.051 (-1.37, -1.17) 
Eligible for free school -7.770 0.265 (-8.29, -7.25) 


meals—missing 


School-level variables 


Special schools -3.041 0.624 (-4.26, -1.81) 
Academy -0.314 0.110 (-0.53, -0.09) 
Girls' school 1.150 0.448 (0.27, 2.03) 
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% pupils with English as an -0.061 0.008 (-0.08, -0.04) 
additional language (EAL) 
Total GCSE points 0.004 0.001 (0.00, 0.01) 
% 5 GCSE A*-C including -0.034 0.004 (-0.04, -0.02) 
English and mathematics 
Intercept -3.585 0.554 (-4.67, -2.49) 


Note: Number of pupils: intervention = 781, comparison = 100,991. Standard deviation of outcome: intervention = 7.44, 
comparison = 11.67. ‘% pupils with English as an additional language (EAL)’, 'Total GCSE points’ and '% 5 GCSE English and 
Mathematics A*—C' all centred using sample mean. 


Table 16: Analysis of mathematics GCSE point score—final model random effects 


WeeVatclalexss Veclalelclacm lane) a 


School 


The primary analysis aimed to test whether pupils who had received mathematics tuition achieved 
significantly higher GCSE mathematics point scores than comparison pupils. Table 12 shows that on 
average pupils who received tuition had slightly higher scores (0.6 points). This difference is 
statistically significant, though small (there are 6 GCSE points to one grade). The standardised effect 
size is 0.051 (95% confidence interval (0.001, 0.101)). 


We also completed a sub-group analysis to test whether the intervention pupils made significantly 
more progress than the matched comparison pupils among a restricted sample of pupils eligible for 
FSM in the past six years. Tables 17 to 20 set out these results. 


Table 17: Sub-group analysis of mathematics GCSE point score—base model fixed effects 


Coefficient Vrelaletclacm-lanel a 95% confidence 
interval 


Intercept 32.361 0.667 (31.054, 33.668) 


Table 18: Sub-group analysis of mathematics GCSE point score—base model random effects 


WEVatclalens) Standard error 


138.863 


Table 19: Sub-group analysis of mathematics GCSE point score—final model fixed effects 


(oxer=ynire(clalt Trelaletslacm)aae) a 95% confidence 
interval 


Pupil-level variables 


‘Intervention 4120 0.621 > (-0.10, 2.34) 
Average Key Stage 3 points— 1.195 0.006 (1.18, 1.21) 
mathematics 
Female -0.193 0.103 (-0.39, 0.01) 

‘Eligible for freeschoolmeas— = = OO. (-8.29, -7.25) 


missing 
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School-level variables 
Special schools -3.442 0.723 (-4.86, -2.02) 


% pupils with special educational -0.007 0.006 (-0.02, -0.00) 
needs (SEN) 


Intercept -6.447 0.445 (-7.32, -5.57) 


Note: Number of pupils: intervention = 220, comparison = 27,195. Standard deviation of outcome: intervention = 7.9, 
comparison = 12.32. ‘% pupils with special educational needs (SEN)’ centred using sample mean. 


Table 20: Sub-group analysis of mathematics GCSE point score—final model random effects 


WVEcVa tc Valens) Jeclalelclacmclanela 


The FSM sub-group analysis identified that pupils receiving tutoring achieved a higher GCSE 
mathematics grade than those in the comparison group, but this difference was not significant (the 
95% confidence interval included zero). The standardised effect size was 0.090 (95% confidence 
interval (-0.008, 0.188)). 


We conducted some further exploratory analysis to look at the impact of important features of the 
mathematics tutoring. We did this by substituting variables reflecting features of the tutoring received 
by pupils for the overall effect of tutoring. This analysis showed that: 


e Each hour of mathematics tuition is associated with an increase of 0.14 in GCSE 
mathematics point score. This was a statistically significant difference (95% confidence 
interval (0.09, 0.18)). Each additional tuition session is associated with a similar increase. 

e Substituting the reasons given for students receiving tutoring for the overall tutoring effect 
results in a significant effect for those with the potential to get grade B or above, but non- 
significant results for those struggling to attain grade D/C borderline and for those at the 
grade D/C borderline who are aiming for grade C. On average, students with the potential to 
get a grade B who received tutoring achieved almost five GCSE mathematics points higher 
than their peers. This was a statistically significant difference (coefficient = 4.53, 95% 
confidence interval (1.43, 7.63)). 


However, given that we have concerns about the accuracy and completeness of the data provided by 
schools on the above variables, this analysis should be treated with considerable caution. 


Analysis of English tutoring 


Tables 21 to 24 set out the results of the main impact analysis for English tutoring. 


Table 21: Analysis of English GCSE point score—base model fixed effects 


Coefficient Teclalelslacm lane) a 95% confidence 
interval 


Intercept (35.298, 37.404) 
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Table 22: Analysis of English GCSE point score—base model random effects 
Variance Standard error 


School 


Table 23: Analysis of English GCSE point score—final model fixed effects 


95% confidence 


fexer-ynice(clalt Teclaletslacmslane) a ‘ 
interval 


Pupil-level variables 


Intervention -1.413 0.584 (-2.56, -0.27) 

Average Key Stage 3 points—English 0.949 0.004 (0.94, 0.96) 

Gender—female 0.926 0.053 (0.82, 1.03) 

Eligible for free school meals —1.235 0.056 (-1.34, -1.13) 

Eligible for free school meals—missing -9.139 0.347 (-9.82, -8.46) 

School-level variables 

Academy -0.587 0.280 (-1.14, -0.04) 

% pupils with English as an additional 0.036 0.008 

language (EAL) (0.02, 0.05) 

% pupils with special educational needs -0.065 0.008 

(SEN) (-0.08, -0.05) 
| Intercept 6.519 0.274 (5.98, 7.06) 


Note: Number of pupils: intervention =409, comparison = 62,970. Standard deviation of outcome: intervention = 5.96, 
comparison = 8.80. ‘% pupils with English as an additional language (EAL)’ and ‘% pupils with special educational needs 
(SEN)’ centred using sample mean. 


Table 24: Analysis of English GCSE point score—final model random effects 


WVEcVatclalexs) Veclaletslaemclage) a 


School 


The primary analysis aimed to test whether pupils who had received English tuition achieved 
significantly higher GCSE English point scores than comparison pupils. Table 15 shows that on 
average pupils who received tuition achieved slightly lower scores (1.4 points). This difference is 
statistically significant, though small (there are 6 GCSE points to one grade). The standardised effect 
size is -0.161 (95% confidence interval (—0.291, -0.03)). 


We also completed a sub-group analysis to test whether the intervention pupils made significantly 
more progress than the matched comparison pupils among a restricted sample of pupils eligible for 
FSM in the past six years. Tables 25 to 28 set out these results. 


Table 25: Sub-group analysis of English GCSE point score—base model fixed effects 


Coefficient SJeclalelclacm lane) a 95% confidence 
interval 


Intercept (34.107, 36.401) 
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Table 26: Sub-group analysis of English GCSE point score—base model random effects 


School 


WVEVarc Valens) 


Pl tclalerclaem-laae)e 


Table 27: Sub-group analysis of English GCSE point score—final model fixed effects 


Pupil-level variables 
Intervention 

Average Key Stage 3 points—English 
Female 

School-level variables 
Academy 

Grammar 

Special schools 

Boys' school 

Girls' school 

Number of GCSE pupils 


% pupils eligible for free school meals 
(FSM) 


% pupils with English as an additional 
language (EAL) 

% pupils with special educational 
needs (SEN) 

Total GCSE points 


% 5 GCSE A*-C including English and 
Mathematics 
Intercept 


Coefficient 


-0.230 
0.934 


0.656 


-0.708 
3.237 
-2.983 
0.099 
-0.742 
0.003 


0.016 


0.042 


-0.035 


-0.004 
0.051 


5.497 


Standard error 


0.89 
0.970 
0.111 


0.354 
1.271 
1.335 
0.691 
0.572 
0.003 


0.017 


0.008 


0.016 


0.003 
0.017 


0.390 


95% confidence interval 


(-1.97, 1.51) 
(0.93, 0.97) 
(0.44, 0.87) 


(-1.40, -0.01) 
(0.75, 5.73) 
(-5.60, -0.37) 
(-1.26, 1.45) 
(-1.86, 0.38) 
(0.00, 0.01) 


(0.02, 0.05) 
(0.03, 0.06) 
(-0.07, 0.00) 


(0.01, 0.00) 
(0.02, 0.08) 


(4.73, 6.26) 


Note: Number of pupils: intervention = 123, comparison = 16,706. Standard deviation of outcome: intervention = 5.68, 
comparison = 9.03. ‘% pupils eligible for free school meals (FSM)’, ‘% pupils with English as an additional language (EAL) ‘, ‘% 
pupils with special educational needs (SEN)’, ‘Total GCSE points’ and ‘% 5 GCSE English and Mathematics 5 A*-C’ centred 


using sample mean. 


School 
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Table 28: Sub-group analysis of English GCSE point score—final model random effects 


The FSM sub-group analysis identified no convincing evidence/significant differences in progress 


between pupils in the intervention and comparison group that were eligible for FSM. The standardised 
effect size is -0.03 (confidence interval (-0.22, 0.17)). 
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Cost 


The Tutor Trust charged £18 per hour for tuition delivered in the Academic Year 2011-2012, although 
there was some scope for negotiation on this, depending on the school’s overall tuition needs. The 
hourly charge was the same irrespective of whether tuition was delivered on a one-to-one, one-to-two 
or one-to-three basis. In the Academic Years 2012—2013 and 2013-2014, the tuition rates changed, 
depending on the ratio of tutors to pupils: one-to-one tuition was charged at £18 per hour; one-to-two 
at £20 per hour, and one-to-three at £26 per hour. Based on groups of three pupils receiving 25 tuition 
sessions, the total cost of the intervention is estimated at approximately £217 per pupil. 
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Process evaluation 


Outcomes 


The process data gathered across Phases 1 to 3 of the evaluation showed that, overall, pupils, school 
staff, and tutors were positive about the impact of the intervention. Note, however, that as the process 
evaluation draws on qualitative data, the findings are not intended to provide a basis for 
generalisation. It is instead about improving understanding of the distinctive nature of key 
stakeholders’ experiences and outlooks. 


Pupil attainment: Teaching staff generally believed that the tuition was contributing to gains in pupil 
attainment, and most of the pupils and tutors reported seeing visible improvements in pupils’ 
performance. However, many teachers were unclear or unsure about the extent to which the tuition 
was having an impact on pupil attainment. These reservations appeared to stem from the fact that 
other intervention work was also being undertaken with the tutored pupils, which made it difficult to 
attribute the impacts to one intervention over another. 


Most interviewees reported that the tuition had helped to develop pupils’ understanding of different 
topics within the relevant GCSE syllabi and had contributed to developing their skills and examination 
technique. One pupil explained: ‘It’s helped a lot. Like, in class we might not have done it yet, but if 
we've done it with the tutor, we've got like an edge on the others.’ A teacher in another school was 
more direct about the impacts on attainment: ‘Out of the 30 pupils who received tuition, 22 got their 
Cs. The eight who didn’t were the ones that didn’t turn up.’ There were no discernible differences in 
interviewees’ views on the effectiveness of the tuition on pupil attainment in mathematics and English. 


Pupil confidence: There was general agreement that the programme had had a positive effect on 
pupils’ confidence in both mathematics and English. Pupils reported gaining confidence in their 
abilities: ‘It’s made me more confident. I’ve got a better overall understanding of the different parts of 
English and mathematics.’ This was largely attributed to the small group environment, which afforded 
tutees the freedom to ask more questions and to expand on their answers: ‘It makes you feel more 
confident—| might not have put my hand up to ask a teacher, but now | might.’ 


Pupil enjoyment: All of the pupils enjoyed the tuition and said they would recommend it to their 
friends.*° One explained: ‘It feels that it is a nicer atmosphere than class: it is more relaxed and you 
can concentrate more.’ Similarly, teachers reported that most pupils enjoyed the tuition, with some 
reporting that their pupils had asked for the tuition to be extended. Some pupils in earlier rounds of 
tuition reported they were enjoying their mathematics and English classes more: ‘I now love 
mathematics!’ However, for most pupils, there was little evidence to suggest the tuition had raised 
their interest in or made them more enthusiastic about mathematics and English beyond their tuition 
session. 


Pupil attendance and behaviour: The evidence was mixed on the impact of the tuition on pupils’ 
attendance and behaviour in school. In some cases, the attendance and behaviour record of the 
targeted pupils was already reported to be good, which was one of the reasons why they were 
selected for the tuition in the first place. Pupils’ behaviour during the tuition sessions was generally 
reported to be good, although there was a high degree of drop out in some cases (see section on 
‘Fidelity’). In the most recent round of tuition, one teacher observed that while his pupils were well 
behaved with their tutor, they continued to ‘play up’ in class. He remarked that it would be interesting 
to see if, in the longer term, this improved behaviour with the tutor would be carried over into the 
classroom. 


23 It should be noted that all of the pupils we interviewed were still receiving the tuition. Unfortunately, it was not possible to 
speak to those pupils who had dropped out. 
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Pupil aspirations: Some pupils and tutors reported that the tutoring had helped to raise pupils’ 
academic aspirations by contributing to improvements in their own knowledge and skills and through 
opportunities to discuss ‘university life’ with a current or recently graduated university student. 


Tutors: The tutors reported they had enjoyed the experience and had found it rewarding. All of the 
tutors consulted would recommend it to their friends, and indeed many reported they had already 
signed up friends to the scheme.“* Tutors viewed the experience as something positive to add to their 
CVs, while many said it had encouraged them to consider a career in teaching or had reinforced pre- 
existing decisions to pursue a career in teaching. Collectively the findings suggest that the Tutor Trust 
will be able to recruit more tutors to the programme should it be scaled up in the future. 


Does tuition benefit all pupils equally? 


There was no clear message from interviewees about the extent to which the tuition impacted on all 
pupils equally. Rather, interviewees felt that the quality of the tutor and their ability to relate to the 
pupils and to teach them appropriately was the cause of variations in impact. While amongst case- 
study schools D/C borderline pupils appeared to be those most frequently targeted for the tuition, 
other groups, including B/C borderline pupils and gifted and talented pupils were also said to benefit. 
Overall, it was the pupils who engaged most with the tuition who were reported to get the most out of 
it, irrespective of academic ability or social background. 


In a minority of cases the tuition was reported to have some unintended consequences or negative 
effects. These largely stem from the way the tuition had been introduced and/or implemented in 
schools. For example, in one school, no mathematics or English teachers had been involved in the 
decision to use the tutors, or in managing their deployment. Instead, the liaison with the Tutor Trust 
and tutors had remained at Senior Leader level, and was not devolved. This had resulted in some 
resentment amongst teachers about the role of the tutors. Similarly, in at least two other schools there 
had been a clash with the timetabling of tuition with after-school revision sessions. This led to some of 
the teachers of those lessons questioning the value and timing of the tuition, preferring instead for the 
pupils to attend their own after-school sessions. Despite this small number of unintended 
consequences, overall, school staff were positive about the tuition. 


Implementation 


Is the service fulfilling an identified need (i.e. high-quality, affordable tutoring)? 


Most of the teachers felt that the tutoring represented good value for money. They all agreed that there was a 
need for affordable tuition in Greater Manchester: ‘Children in our area do not have the money to have this kind 
of tuition outside of school’. Most agreed that the tuition from the Tutor Trust could help meet this need. Where 
teachers expressed reservations about its value for money, this was associated with limitations in how the 
tutoring had been implemented in schools. For example, the tutoring would have represented better value for 
money had the costs been spread amongst more pupils in the schools where drop out had been a problem. In 
addition, while case-study schools appeared to like the perceived light-touch nature of their involvement in 
coordinating the tutoring in schools, some teachers suggested their schools may have leveraged greater value 
from the tutoring had they invested more time in the programme: ‘Ideally the school would invest more time in 
coordinating it but not all teachers have this time to give’. 


Why do some schools decide not to take up the offer of tuition? 


The tutoring programme was largely attractive to schools, evident in the number of schools who 
request tuition year on year. Most school staff reported they valued the professionalism of the tutors 
and flexibility provided by the scheme. However, some schools chose not to take the Tutor Trust up 
on their offer of tuition. We collected responses via email and telephone from a total of ten such 


4 It should be noted that all of the tutors we interviewed were still delivering the tuition. We did not speak to any tutors who had 
dropped out of the programme. 
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schools between 2012 and 2014. The main reasons cited for why these schools had decided not to 
purchase the Tutor Trust tuition included: 


e it was too expensive 

e schools already had successful tuition programmes in place 

e schools had had bad experiences of tuition programmes in the past 

e concern that the Tutor Trust would be unable to deliver at a level that was suitable for their 
D/C borderline pupils. Their perception was that the tuition was aimed solely at more able 
learners. 


Despite this, most of the schools reported they would consider paying for Tutor Trust tuition in the 
future. Some schools reported that offering the tuition at lower cost would make the Tutor Trust’s offer 
more attractive to their schools, while others reported they would like more information from the Tutor 
Trust about the intervention, including evidence of impact in other similar schools. 


Are the tutors ‘high quality’? 


In most cases, the school staff we spoke to reported that the quality of tutors was generally high and 
that the quality of tutors greatly influenced the impact of the programme. For teaching staff, ‘high 
quality’ meant possessing the relevant pedagogical skills; being able to engage and interact 
successfully with pupils and target sessions appropriately; having good subject and curriculum 
knowledge; and being committed, reliable, flexible to changing needs, and willing to work with a range 
of different pupils. The Tutor Trust advertises its tutors as high quality; indeed, each tutor is subject to 
a recruitment process, a training programme, and ongoing support and (in the last year) quality 
assurance workshops. However, during eight of the 11 visits to schools senior managers raised 
concerns about tutor quality. Issues related to tutors’ knowledge of the curriculum, their ability to 
relate to and engage their tutees, and their ability to tutor and ‘teach’ the pupils. These findings should 
be seen in the context of a generally positive reaction from teachers to the quality of tutors; where 
problems were reported, these were in relation to a small minority of tutors. All of the schools in which 
interviews took place in Phase 3 had replaced at least one tutor during their involvement with the 
programme. The requests to change tutors were attributed to a range of issues, including tutors 
having poor English language skills, acting unprofessionally, or being late for sessions. Where tutors 
did not meet the expected quality standards, schools felt that this could have negative impacts on 
their tutees. 


Are tutors delivering the tuition sessions in line with schools’ expectations? 


The tuition sessions were generally delivered in line with schools’ expectations. Schools were able to 
prescribe the frequency and the timing of the tuition, and the Tutor Trust appear to have responded 
appropriately to these requests. 


By Phase 3, the Tutor Trust had expanded significantly, and schools reported that the Trust had no 
difficulty in meeting the demand for tuition, clearly having the capacity to provide a number of tutors 
who were flexible over their availability. 


The tutors have been deployed in different ways in different schools and staff like the flexibility offered 
by the scheme: ‘The tutoring is flexible. You can discuss and set up a package that that suits you. 
This is useful because different schools have different needs’. The fact that most tutors were happy to 
accommodate schools’ changing needs and requirements was viewed as a strength of the 
programme. 
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Is the Tutor Trust meeting its recruitment targets? 


At the inception of the evaluation, the Tutor Trust provisionally intended to work with 50 schools in 
Phase 3. It nearly met this target, working with 46 schools in Phase 3, according to its internal 
records. Over the course of the evaluation the Tutor Trust reports that it worked with 55 schools. This 
demonstrates the Tutor Trust’s ability to recruit schools and to deliver tuition at volume. 


There is also evidence to suggest that the Tutor Trust has been successful in recruiting secondary 
schools in deprived communities. 58% of the Tutor Trust secondary schools who provided data for the 
impact analysis fell into the highest 20% of schools for their FSM eligibility. 


Barriers to delivery 
School staff reported the following barriers to delivery: 


e Some schools that chose to provide after-school tuition experienced a drop off in pupil 
attendance. This attrition was due to pupils prioritising other subjects as it got closer to the 
exam period (e.g. drama final assessment; art portfolio); pupils’ school attendance dropping 
off in general; and some pupils from some of the schools stopped attending tuition as they felt 
it was of little benefit. 

e In the final phase of the evaluation, unwillingness of some tutors to work with pupils across 
the range of abilities (specifically, the least able). In at least one school this restricted the 
types of pupils that could be offered tuition. 

e Not having enough time to properly brief the tutors and/or collect feedback from the tutors 
following the tuition sessions. 

e Variations in tutor quality: ‘My impression is that some of the tutors spent time on things they 
felt comfortable doing, rather than on what the pupils needed to cover.’ Also see ‘Fidelity’, 
below). 


Key factors for success 


From the qualitative data, the Tutor Trust tuition is likely to be most successful in bringing about an 
increase in attainment when: 


e the tutors are high quality and able to respond appropriately to and interact successfully with 
their tutees. Tutors need to be committed, reliable, flexible to changing needs, and willing to 
work with a range of different pupils. Where tutors are of poorer quality, this can have 
negative impacts on their tutees. 

e the tuition is scheduled at times most convenient for staff and pupils (e.g. earlier in the year, 
before the GCSE examinations). In some schools, providing tutoring during the school day, 
rather than after school, may have reduced attrition: ‘The pupils wanted to have help although 
the reality was that many weren't prepared to stay behind after school to be helped’. Clashes 
with popular and/or important after-school activities should be avoided. 

e English and mathematics teachers are involved in the planning and management of tuition so 
that it is not viewed as a ‘bolt-on’, or unrelated to their classroom teaching. The tuition 
appeared to work best where class teachers (or Heads of Department) had been fully 
involved in its design and implementation. It is important that liaison with tutors and the Tutor 
Trust is devolved to staff who can ensure that the tuition is targeted at those pupils that stand 
to benefit most, and ensure that tuition is better integrated with class provision. 

e school time is continually invested in working closely with their tutors, monitoring progress 
and establishing feedback mechanisms. 

e there is continuity in the tutor—tutee relationship throughout the tutoring period. Most of the 
teachers and tutors interviewed reported that it can take several weeks for a tutor to build a 
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relationship with a pupil, to earn their trust, and to diagnose and understand their learning 
needs. Schools that hosted a number of different tutors with the same child did not appear to 
gain the same benefit from the programme as those which retained the same tutors over the 
course of the intervention. 


Fidelity 


The Tutor Trust model of tutoring is flexible by design, and schools are free to decide how best to 
deploy the tutors with their pupils. They are also free to choose the year group(s) and support needs 
of the pupils receiving the tutoring. All of the case-study schools intended to use either one-to-two or 
one-to-three support. Approximately half of the case-study schools timetabled the tuition to take place 
within the school day, with the other half choosing to run it after school. The evaluation focused on the 
impacts of the tutoring on Year 11 pupils (although other year groups may have also received support 
from a tutor). 


Given the parameters of the intervention described above, both the process and impact evaluation 
revealed that the deployment of tutors was rather more fluid than was anticipated. For example, while 
most schools had expressed a preference for tutors to work with groups of two or three pupils, some 
tutors reported they had worked with pupils on a one-to-one basis due to other pupils not turning up to 
their sessions. This appeared to be particularly the case for after-school sessions and sessions 
scheduled close to Year 11 examinations, where attrition was greater. In at least one school, tutors 
had been deployed in a wider range of roles, including to support marking and assessment (away 
from pupils) and to provide in-classroom support in a role more akin to that of a teaching assistant. 


A small scale but ongoing issue with programme fidelity concerns the quality of tuition. Given that the 
quality of the tutor and tuition is considered key to leading to impact it is imperative that continued 
attention is given to ensuring all tutors are high quality. 


Formative findings 
How could the programme be improved? 


e The Tutor Trust should continue to address variations in tutor quality to ensure that all tutors 
are up to the required standard of conduct and performance. Building in additional quality 
audits, or establishing more formal feedback or performance reviews would be welcomed by 
schools and tutors alike. 

e Schools should consider better utilising subject teacher time to get more out of the tuition. 
Focusing on connecting the tutor sessions more effectively with classroom activities and 
pupils’ areas for development would be beneficial. 

e Schools might usefully invest more time in setting up systems to monitor the content and 
progress made in tuition sessions. 

e Schools might better utilise the opportunity to liaise with tutors at the initial tutor meeting 
(suggested by the Tutor Trust). Such ‘start-up’ or ‘get to know you’ events enable staff and 
pupils to meet tutors and begin to build relationships. 


The Tutor Trust should continue to provide good practice guidance to schools with practical steps for 
how schools can make the most out of tuition. 
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Conclusion 


Key conclusions 

1. Due to the limitations of the study design and the absence of a high-quality comparison 
group, this evaluation has not provided a secure estimate of the impact of the project on 
academic outcomes. 

2. Participating pupils achieved slightly higher mathematics GCSE scores than pupils in the 
comparison group, and lower English GCSE scores than pupils in the comparison group. 
However, it is not possible to attribute either change to the tuition provided. 

3. Schools involved in the qualitative interviews were positive about the tuition, keen to work 


with the Tutor Trust again, and largely confident that the tuition was beneficial for their 
pupils. 


4. Teachers believed that there was a need for more affordable, high-quality tuition, and that 
the Tutor Trust helped meet this need. 


5. It is recommended that the Tutor Trust continues to monitor tutor performance and identifies 
mechanisms to increase the consistency of tuition. Schools should be active participants in 
this process and classroom teachers need to be involved in the planning and management 
of tuition to ensure that in all cases tuition complements work in the classroom. 


Limitations 


In order to attribute any effect directly to the study, we would have carried out a randomised controlled 
trial (RCT). As this was not possible, we cannot attribute any effect to the tutoring. In an RCT the 
control group is an ideal estimate of what would have happened on average if the intervention group 
had not received the intervention. All the observed and, crucially, the unobserved characteristics of 
the two groups will be balanced on average at the start of the trial. This means that when outcomes 
are measured, the differences can be attributed to the intervention. Interpreting the model relies on 
the assumption that everything that matters for pupil outcomes is balanced on average between the 
two groups at baseline. Since receipt of tuition was not decided by random allocation, if unobserved 
characteristics are a determining factor of whether the pupil gets tutoring and also correlate with test 
score outcomes, then the impact estimates could be biased, and the extent of the bias is unknown. 
Note that in a deviation from the original planned analysis, we were unable to control a number of 
potentially influential variables. These comprised: pupil-level special educational needs (SEN); 
English as an additional language (EAL); ethnicity; and looked after children (LAC) status. These 
variables were excluded from the final analysis due to a change in the level of consent required by the 
NPD in order to release this data, after we had completed our data collection in schools. Failing to 
account for important variables (Known and/or unobserved) has been shown to result in spurious 
effects, certainly those of a magnitude we see here (Coe, 2009). 


The analysis is also based on (often retrospective) school-level data. The process evaluation revealed 
that some schools did not keep accurate records of their tuition sessions (e.g. number of sessions per 
pupil, the pupil-tutor ratio). It is therefore likely that the data provided by schools may not be an 
accurate record of the tuition that actually took place. 


The impact analysis is based on data from 22 schools for mathematics and 17 schools for English— 
lower than the 50 schools on which our power calculation was based. The number of pupils included 
in the analysis was lower than anticipated, but sufficient to detect a statistically significant effect for 
the all-pupils analysis. However, despite this, it was not possible to provide a secure estimate of the 
programme’s impact on pupil attainment due to the weaknesses noted above. In addition, the lower 
pupil numbers reduced the likelihood that the evaluation would be able to detect an effect of a similar 
size for the subset of pupils who were eligible for FSM. 


The Tutor Trust model of tuition provision could be extended to other areas, although geographical 
proximity to a university or college could be a limiting factor. 
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Interpretation 


Due to the limitations of the study design noted above, this evaluation has not provided a secure 
estimate of the impact of the Tutor Trust on academic attainment or made a substantial contribution to 
the existing evidence bases on one-to-one or small group tuition. It is also important to recognise that 
our evaluation did not seek to assess the quality of individual tutors or, indeed, to identify any 
differential effect of different group sizes within the impact analysis. More granular research into the 
different models of tutoring, to include data on tutor quality and group size, may provide more 
illuminating and definitive evidence. 


While acknowledging the methodological limitations of the study, it is not clear from our evaluation 
why the tuition appears to have been effective in improving pupil outcomes in mathematics, but not in 
English. There are a number of possible reasons: 


e unmeasured or unobserved differences in the implementation of the tuition (e.g. quality of the 
tutors, one-to-one or small group tuition, group sizes, number of hours or sessions, after- 
school sessions with a greater drop-out in attendance). 

e unmeasured or unobserved differences in the pupils selected for tuition (e.g. motivational 
differences, differences in selection criteria for mathematics and English, whether students 
were selected based on how much the school perceived they would benefit from English 
tuition or for ‘other’ reasons). Selection was not random and it is easy to conceive that the 
method of selection is correlated with the outcome. 

e some schools told us their pupils took their English GCSE early in Year 10. It was not 
possible to identify early GCSE takers from the data alone.”° If this was a common 
occurrence for English and not mathematics (a higher proportion of Year 10 pupils received 
tuition in English), it is not unreasonable to expect to see lower grades than had the GCSEs 
been taken at the end of Year 11. 

e the inherent nature of mathematics compared to English as a subject. If the mathematics 
tuition is more closely aligned to the mathematics outcome measurement (GCSE attainment) 
than English tuition is to the English GCSE assessment, then it is more likely to see a 
stronger relationship between intervention and outcome in mathematics. 


The process evaluation has provided useful insights. Generally, teachers in participating schools were 
positive about the tuition provided across both subjects and a large majority of the senior leaders we 
interviewed were keen to work with the Tutor Trust again. Moreover, school staff reported that the 
tuition offered good value for money and believed that it was contributing to positive impacts for their 
pupils. It is evident that the Tutor Trust has repeat customers, perhaps testament to their conviction of 
its worth. The teaching staff we interviewed generally believed that the tuition was contributing to 
gains in pupil attainment, and most of the pupils and tutors reported seeing visible improvements in 
pupils’ performance. However, many teachers were unclear or unsure about the extent to which the 
tuition was having an impact on pupil attainment, largely due to other intervention work being 
undertaken with the tutored pupils, which made it difficult to attribute the impacts to one intervention 
over another. 


The findings from the process evaluation suggest the Tutor Trust tuition was perceived to impact most 
successfully on ‘softer’ outcomes such as pupils’ confidence in mathematics and English, and on their 
aspirations to continue on to further study. However, this in large part hinges on the quality of the 
tutors and the way the tuition is managed and delivered in school. As other research has suggested, 
when the right design and administrative structures are implemented, volunteer tutoring programmes 
can be effective (Tepper Jacob et al., 2014). The Tutor Trust need to continue their efforts to ensure 
that only the best quality tutors make it through their recruitment and training programme. At the same 


oa All GCSE results, including those taken early, are given at the end of Key Stage 4. Year 10 pupils in our datasets could be 
early GCSE takers or they could be pupils receiving tuition in Year 10 in readiness for Year 11. 
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time, schools can help to drive up tutor quality by liaising more closely with their tutors, and providing 
more instruction and feedback. If the improvements set out in the process section of this report 
continue to be implemented, and schools communicate more closely with their tutors; devolve 
responsibility for those who are running the tuition in school to the subject teachers themselves; and 
better integrate the tutoring into the pupils’ classroom learning, then we would hypothesise that the 
tuition might have a greater effect. 


Future research and publications 


Further research is needed to be able to determine the effect of the tuition. Ideally the research 
design would take the form of an RCT, to attribute any effect to the tuition. It would draw on a larger 
sample of pupils who were eligible for FSM, so that the effects on FSM pupils could be robustly 
measured. The findings from the process evaluation suggest that the tuition is having an impact on 
‘softer’ outcomes, such as pupils’ subject-level confidence and aspirations to continue on to further 
study. It would be useful to explore these softer impacts in more detail as part of any future analysis. 
One of the strengths of the Tutor Trust model (from schools’ perspectives) is its flexibility. However, 
differences in implementation may lead to differing effects and it might be useful to explore the effects 
of different models of delivery, such as the relative impact of different group sizes, level of integration 
of tuition with learning and teaching within the school, quality of tuition, and whether tuition was in 
addition to or a part replacement for subject lessons. Drawing on the learning from this evaluation, 
securing the buy-in of schools to participate in the evaluation will be crucial. A cost effectiveness 
analysis would also be a beneficial addition to future research. 
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Appendix 1: Consent statement in Tutor Trust contracts 
with participating schools 


Use of pupil data 


The School shall assist with the independent monitoring and evaluation of the Tutor Trust’s work, 
which is being carried out by the National Federation for Educational Research (NFER) on behalf of 
the Education Endowment Foundation (EEF). This includes providing reasonable assistance to the 
University of Durham which is evaluating the impact of all of the work funded by the EEF (or any other 
third party that may be appointed to evaluate the EEF’s work from time to time). 


The School shall at its own cost: 


e complete pupil data forms in the format requested by the Tutor Trust providing complete and 
accurate data on all pupils nominated to receive tuition services (irrespective of whether 
pupils go on to receive tuition) (the Pupil Data) by agreed deadlines; and 

e accommodate any reasonable requests for case study visits from NFER (or any other third 
party that may be appointed to evaluate the EEF’s work from time to time) for their evaluation. 


In addition to the right of Tutor Trust to use and process any Pupil Data or other data provided by the 
School or any pupil from time to time for the purposes of providing the Services, the School grants its 
consent for all Pupil Data and or other data obtained by the Tutor Trust from time to time in the course 
of providing the Services to be: 


e disclosed to third parties for analytical and statistical purposes from time to time (including the 
NFER, the University of Durham, the EEF and any other evaluators EEF may appoint from 
time to time); and 

e stored in an anonymised form and made available for public access on the UK Data Archive. 


The Pupil Data will be linked to other datasets such as the national Pupil Database (held by the 
Department for Education). This linking is solely for statistical purposes and no information that could 
identity individual pupils or schools will be made public. Only aggregated data will be used in 
evaluation outputs, such as reports or presentations. 


The Tutor Trust shall keep the Pupil Data confidential and shall use reasonable commercial 
endeavours to procure that any third parties to whom Pupil Data is disclosed pursuant to clause 0 is 
kept confidential prior to such time as it is anonymised. Notwithstanding the foregoing, the Tutor Trust 
may disclose the Pupil Data: 


e as permitted under clause 0; 

e to its employees, officers, representatives, advisers, agents or subcontractors who need to 
know such Pupil Data for the purposes of carrying out the Tutor Trust's obligations under this 
Agreement; 

e as may be required by law, court order or any governmental or regulatory authority. 
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Appendix 2: Example consent letters and interview 
introductions 


Example of a consent letter for participating schools 
Dear [insert name] 


As you are aware, the National Foundation for Educational Research (NFER) is evaluating the 
mathematics and/or English tuition currently being delivered to some Year 11 pupils at your school by 
the Tutor Trust. 


| understand that the Tutor Trust has made you aware of the evaluation, and of the fact that we may 
be contacting you to request that you help us with this work. 


What will participation involve? 


Participating in the evaluation would involve a researcher making a one-off visit to some schools who 
are receiving tutoring to speak to: 

you, as the lead contact for the Tutor Trust 

one teacher whose Year 11 pupil(s) have received tutoring 

one tutor (if present at the time of the visit) 

one young person in Year 11 who has received tutoring, OR a pair of young people in Year 11 


who have received tutoring. 


We would be looking to visit each school in May 2014. Each visit will take no more than half a day. 
We are happy to discuss how much of the above agenda is feasible, however. 


Please be assured that all the information collected as part of the visits will be treated in strict 
confidence and according to the rules laid down by the Data Protection Act and NFER’s Code of 
Practice. No school or individual will be identified in any research publication resulting from this work. 


Visiting your school would allow us to gain valuable information which will inform tuition delivery in 
future so that it can benefit other young people. 


We will contact you soon to discuss the possibility of your school participating in the evaluation. In the 
meantime, please do not hesitate to contact me if you wish to discuss any aspect of the evaluation. 
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Example of a consent letter for non-participating schools 
Dear [insert name], 


NFER is currently evaluating the impact of Tutor Trust tuition on behalf of the Education Endowment 
Foundation (EEF). As part of the evaluation, we are keen to hear from schools such as yours, which 
have chosen not to buy tuition from the Tutor Trust in this academic year. 


| would be extremely grateful if you would be willing to have a very brief telephone conversation with 
me about the reasons for your school’s decision not to buy Tutor Trust tuition this year. This would 
only take approximately 10 minutes, and would be conducted at a time of your convenience. 


Please be assured that all the information that you provide during our conversation will be treated in 
strict confidence and according to the rules laid down by the Data Protection Act and NFER’s Code of 
Practice. We will not identify any school or individual in any report arising from the research. 


My colleague Neelam Basi will contact you by telephone shortly to discuss booking the call. In the 
meantime, please do not hesitate to contact me on xxx if you wish to discuss any aspect of this work. 
In any event, | look forward to speaking with you soon. 


Example wording used at the start of interview with headteachers 


Thank you for agreeing to be interviewed and for your support for this evaluation. As you are aware, 
NFER is currently evaluating the effectiveness and impact of Tutor Trust tuition, on behalf of the 
Education Endowment Foundation. 


As part of the evaluation we are carrying out case studies of schools that are receiving tuition from the 
Tutor Trust in this academic year. The purpose of this interview is to explore why you chose to buy 
tutoring, and your perceptions of the quality and impact of Tutor Trust tuition. 


The information that you provide will be treated entirely confidentially and anonymously. We will not 
identify any individual or school in the outputs from our evaluation. 


The interview will take up to 60 minutes. | would like to record it, to supplement my notes. The 
recording will be deleted once | have taken notes from it. Are you happy for me to do this? 


Is there anything that you would like to ask me about the evaluation? 
Example wording used at the start of interview with teachers 


Thank you for agreeing to be interviewed and for your support for this evaluation. As you are aware, 
NFER is currently evaluating the effectiveness and impact of Tutor Trust tuition, on behalf of the 
Education Endowment Foundation. 


As part of the evaluation we are carrying out case studies of schools that have chosen to buy tuition 
from the Tutor Trust in this academic year. The purpose of this interview is to explore your 
perceptions of the quality and impact of Tutor Trust tuition. We understand that one of more of your 
students has received Tutor Trust tutoring, is that right? 


If yes... (if no, probe and end interview if none of teacher’s students has been tutored) 


The information that you provide will be treated entirely confidentially and anonymously. We will not 
identify any individual or school in our evaluation outputs. 


The interview will take up to 40 minutes. | would like to record it, to supplement my notes. The 
recording will be deleted once | have taken notes from it. Are you happy for me to do this? 
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Is there anything that you would like to ask me about the evaluation? 


Example wording used at the start of interview with pupils 
My name is XXXX, 


I’m a researcher for the National Foundation for Educational Research. I’d like to talk to you about the 
tuition that you’ve been getting from [name of tutor]. This will help me to understand how well the 
Tutor Trust tuition is working and if there’s any way to improve it. There are no right or wrong answers 
to the questions | will ask you. The interview will take up to 30 minutes (40 minutes if paired 
interview). 


All the information you give me in this interview will be treated confidentially within our research team. 
We will use your information to inform a research report into how effective the Tutor Trust is, but won't 
name you in the report. The report will go to the Education Endowment Fund and the Tutor Trust. 


Are you happy for me to interview you? We can stop the interview at any time you like. Do you mind if 
| record this conversation? | will delete the recoding once | have typed up my notes. 


Is there anything you would like to ask me before we start? 
Example wording used at the start of interview with tutors 


Thank you for agreeing to be interviewed and for your support for this evaluation. As you are aware, 
NFER is currently evaluating the effectiveness and impact of Tutor Trust tuition, on behalf of the 
Education Endowment Foundation. We’re in the third year of our evaluation so we also spoke to some 
tutors in 2012 and 2013. 


As part of the evaluation we are carrying out case studies of schools that are receiving tuition from the 
Tutor Trust in this academic year. The purpose of this interview is to explore your perceptions of the 
programme and the training you have received. 


The information that you provide will be treated entirely confidentially and anonymously. We will not 
identify any individual or school in any of our reports. 


The interview will take up to 40 minutes. | would like to record it, to supplement my notes. The 
recording will be deleted once | have taken notes from it. Are you happy for me to do this? 


Is there anything that you would like to ask me about the evaluation? 
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Appendix 3: Example pupil data form 


Tutor Trust Secondary Evaluation Pupil Data Form 
Academic Year: 2013-2014 


Name of School 


School postcode 


DfE number 


Please use this form to record data on all pupils who received Tutor Trust tuition in mathematics and English during the academic year 2013-2014, as well as 
those who were identified to receive tuition but did not actually receive it. 
Please complete and return this form, with password protection, to the Tutor Trust by Friday 27th June 2014. 

Part 1: Pupil information Part 2: Tuition information 


Pupil's name Unique Date of Birth Looked In which Reason for Reason for Tuition sessions and Tutor:pupil Did pupil miss Tuition sessions and Tutor:Pup Did pupil 


Pupil After subjects tuition: tuition: hours: MATHEMATICS ratio: lessons to hours: ENGLISH il ratio: miss 
ENGLISH lessons to 


Number Child did you MATHEMATI ENGLISH MATHEMATICS receive 
(UPN) intend this (ej 
pupil to 
receive 
tuition in? 


receive 
tuition?: 
ENGLISH 


tuition?: 
MATHEMATICS 


First name Second name Family or Please Please Please add in Please add Total Total From the drop- Please select Total Total From the Please 
(personal) (personal) if surname 00 select select the the main in the number of number of down list, select 'Yes' or 'No' number of number of drop- select 'Yes' 
applicable 'Yes' or appropriat reason why main mathemat | hours of the tutor—pupil from the drop- English hours of down list, or 'No' 
‘No' e you selected reason ics tuition mathemat | ratio for the down list to tuition English select the from the 
from subject(s) this pupil to why you sessions ics tuition mathematics indicate sessions tuition tutor— drop- 
the from the receive selected attended received in tuition sessions whether each attended received in pupil ratio down list 
drop- drop-down mathematics } this pupil in academic that the pupil pupil habitually in academic for the to indicate 
down list. tuition. to receive academic year took part in. For | missed a lesson academic year 2013- | English whether 
list to English year 2013- example, if a to receive year 2013-— | 2014 tuition each pupil 
indicate tuition. 2013- 2014 tutor delivered mathematics 2014 sessions habitually 
whether 2014 tuition to two tuition that the missed a 
each pupils, select pupil took lesson to 
pupil is ieee part in. receive 
looked- For English 
after or example, tuition 
not. if a tutor 
delivered 
tuition to 
two 
pupils, 


select '1:2' 


Tutor Trust: Secondary 


Appendix 4: Total number of Mathematics and English 


tuition sessions 


Table A1: Total number of Mathematics tuition sessions delivered to each pupil 


Number of tuition 


sessions received 


Oo SO NN) |) oO] B) WO! MY BF 


NIL MIEN! NM) NI NM) BIB) BIB) RB RP eB 
oO; N) ao} B) NS] 0] &) om] ow!) N| BIO 


wo 
o 


Total 


Education Endowment Foundation 


Phase 1 


242 


Phase 2 


323 


Phase 3 


53 


20 


19 


25 


233 


44 
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Table A2: Total number of English tuition sessions delivered to each pupil 


LUT aay ey=yaxe)maeliaceyal 


Boca ar aleacoiicd Phase 1 Phase 2 Phase 3 
1 2 7 35 
2 4 35 
3 5 
4 1 11 
5 10 62 17 
6 9 41 
it 12 
8 17 6 
9 11 
10 3 23 83 
12 26 31 
13 3 
15 33 
16 1 
17 22 
25 19 
30 21 
34 1 
36 1 
40 1 
52 1 
58 1 
Missing 2 95 
Total 74 275 339 
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Appendix 5: Security classification of trial findings 


1.Design: ETower 3. Attrition: A Balance: 5. Threats to 


What is the soi What is the Adjustment voles 8 
Adjustment to 


anlialianielans : 
AV-) Me} de] ge) 8) to rating to 
detectable account for issues 


quality of the 


design of the rolUimigelsimagl-) FYelele] 0} a) ai fe) 
i Eile) ay SE ME LO tion? oo at 
| H V : : 
tbat start? hs interpretation? 


Evaluation design Implementation Analysis and interpretation 


Fair and clear experimental Well-balanced on 
design (RCT) : observables 


No threats to validity 


Fair and clear experimental 
design (RCT, RDD) 


Well-matched comparison 
(quasi-experiment) 


Matched comparison 
(quasi-experiment) 


< 0.6 < 50% 
0.6 


Vv 
oa 
No comparator > > 50% Significant threats 


The final security rating for this trial is 0 @. This means that the conclusions have very low security. 


e This evaluation was designed to have a matched control group, and therefore could have achieved a 
maximum of 3 &. 

e The level of attrition was moderate—resulting in 3 @ for this criterion. 

e The MDES is large, corresponding to a rating of 2 @. 

e The matching was not optimal; in particular, the evaluators were unable to include pupil-level SEN, 
looked-after status, and EAL data in the matching or the statistical modelling, as they had planned. It 
was difficult to account for the pupil-level selection criteria in the analysis; in particular, pupils may have 
been selected because they were underperforming. Therefore, the design that the evaluators were able 
to execute is classified as ‘poor’ and awarded 1 @. 

e Balance at baseline was high for many salient characteristics. However, for both the mathematics and 
the English analyses the control pupils had higher prior attainment scores. These differences were large 
(ES in the order of 0.2—0.5), and while recognising that prior attainment was included in the analysis, 1 
@ is deducted from the overall score for this reasons. 

e There were no major threats to validity. 

e In conclusion, the overall security rating is O @. 
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Appendix 6: Cost rating 


Cost rating Description 


Very low: less than £80 per pupil per year. 


Low: up to about £200 per pupil per year. 


Moderate: up to about £700 per pupil per 
year. 


High: up to £1,200 per pupil per year. 


Very high: over £1,200 per pupil per year. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/2 
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Where we have identified any third party copyright information you will need to obtain permission 
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